Device and method for selective recall and preservation of events prior to decision to record the events

ABSTRACT

A device and process for event-activated recording generally includes the steps of providing an acquisition buffer having at least one finite extent of a recording medium. Events are recorded in the acquisition buffer. Current events in place of earlier events are then recorded within the acquisition buffer. Action by the user or triggering system activates the preservation of all or part of the record of the most recent events recorded in the acquisition buffer upon the detection of one or more predetermined events. A portion of the record of events selected by the user is preserved by reserving a portion of the record of events from being overwritten by other events, and/or by transferring the selected portions of the record of events to a predetermined permanent location. Portable devices and applications to the recording telephone conversations are also disclosed, particularly a method of producing a record of a telephone conversation on the consent of appropriate parties, which consent may be secured before, during or after such conversation.

FIELD OF THE INVENTION

This invention relates to the mechanical, electrical, magnetic or othermechanized recording and reproduction of auditory and/or visual or otherdata, and particularly to improvements to a process of recording whichenables human users to decide whether to preserve a record of eventsthat have already transpired.

BACKGROUND OF THE INVENTION

My earlier patent, U.S. Pat. No. 5,845,240 (Selective Recall andPreservation of Continuously Recorded Data), discloses a recordingmethod that effectively allows a user to decide to record an event afterthe event has taken place. This is achieved by continuously recordingall ambient events in a continuous logical loop on a finite extent ofrecording medium and allowing users to select for permanent preservationportions of the material thus recorded, before they are overwritten withnew material.

The present invention comprises improvements to and applications of thisbasic technology.

One area of technology addressed by the present invention isevent-activated recording, in which a recording process is initiated onthe occurrence of some event that is recognized by some mechanism,algorithm or other automated means. Current examples include:

(1) Voice-activated recording of speech

(2) Sound-activated recording of sound

(3) In a security system, audio and/or video recording activated ondetection of movement

-   (4) In a medical instrument, recording initiated by some significant    event, e.g., an arrythmia

A drawback common to all of these methods is the fact that they do notrecord the often-significant events leading up to the event thattriggers the recording process. (Indeed, because some time elapsesbetween the occurrence of the triggering event, detection of the event,processing of the resulting signal, recognition of the event assignificant, and finally initiation of the recording process, thetriggering event is usually not recorded, or only partially recorded.)

The present invention overcomes this drawback by providing anacquisition buffer (of a rewritable recording medium) in which currentevents are continuously recorded. When the recording process fills theacquisition buffer, it overwrites the record of the earliest eventsrecorded in the buffer with current events. On detection of a triggeringevent, a predetermined portion of the current contents of theacquisition buffer (which may be the buffer's entire contents, or maycorrespond to a time interval that a user may specify) is transferred tothe (permanent) recording medium normally used for event-activatedrecording. This has the effect of permanently preserving the eventsleading up to the triggering event.

A further area of application of the present invention is telephony.Current methods of recording telephone conversations share the drawbacksof conventional methods of recording: a recording process must beinitiated (and, when applicable, consented to by parties to theconversation) before any of the events to be recorded have transpired.

Accordingly, it is an objective of the present invention to enable theeffective recording of portions of a telephone conversation after thoseportions have taken place. It is a further objective of the presentinvention to secure the consent of parties to the conversation as acondition to such recording, or to making such recording accessible.

Current laws generally require the consent (in most cases, the priorconsent) of parties to a telephone conversation as a condition to recordthe conversation. However, that consent may not have been given prior toa conversation which a party later desires to have had recorded.Accordingly, it is an objective of the present invention to compile aninaccessible record of a telephone conversation and to make that recordaccessible to appropriate parties on the (possibly later) consent ofappropriate parties.

It is a further objective of the present invention to enable therecording of different kinds of data on interchangeable media such asflash memory.

Portable recording devices constitute a further area of application ofthe present invention. Current portable recording devices share thedrawbacks of conventional methods of recording: a recording process mustbe initiated before any of the events to be recorded have transpired.The present invention is embodies in two devices that overcome thesedrawbacks: a pocket device and a wrist recorder, both advantageouslyusing interchangeable media such as flash memory.

SUMMARY OF THE INVENTION

The present invention differs enables users effectively to record wordsthat have already been spoken, or events that have already taken place,prior to the time that a user decides to record them. This is achievedby recording events on a finite extent of recording medium in acontinuous logical loop (i.e., overwriting the earliest recorded datawith new data whenever the end of the medium used for this continuousrecording is reached), and playing back and/or permanently preserving(capturing) portions of the recorded data as selected by the user.

A further advantage of this invention is that it allows users topreserve the spontaneity of their speech and actions by dispelling theself-consciousness that sets in when people know that they are beingrecorded-which happens even when people record themselves, triggered bythe very act of switching on a recording device. Moreover: Even incontinuous monitoring methods, where continuous recording on anindeterminate supply of recording medium is practicable, there is theneed to go back to search for and edit out the parts to be preserved,discarding the rest. The present invention goes far towards eliminatingthat need, by capturing data permanently only in response to a user'sexplicit request (although this request must come within apredetermined, finite period from the time of the event).

The present invention may be applied to sound or video recording, aswell as recording

(1) the sequence of user actions (or other events) on a computer orother machine or on a musical instrument,

(2) streams of data produced by medical or weather instruments, or

(3) streams of data produced by any broadcasting means, including“netcasting” over the Internet.

The invention may be embodied, with varying degrees of practicality,using any rewritable recording medium, including electronic memory,tape, wire, or discs, both analog and digital. However, the currentpreferred medium for continuous recording is digital memory, as itentails a minimum of wear, power consumption and bulk—especiallyimportant in a portable device.

The essential components of this invention—beyond a minimum ofappropriate hardware to detect, acquire and preserve data—arealgorithms, which may be implemented as either hardware or software orboth. The invention may be implemented as a program on a general-purposecomputer, using either a single-thread or a multithreaded operatingsystem, as long as means of acquiring and playing back data areprovided.

Following is a description of an embodiment of this invention as acomputer type program to capture audio data, running on aninterrupt-driven computer of known type and using digital memory as thecontinuous recording medium. The same scheme may be used in a dedicateddevice other than a general-purpose computer; and variations of thisscheme may be used to record other streams of events, e.g., video data.In this last case, the division of the continuous recording into blocksseparated by intervals of silence may be replaced by a separation intoblocks each having the same base frame as detected by a videocompression algorithm.

This embodiment uses, in place of a single circular acquisition bufferfor storing continuously recorded data, a circular array of smaller,separately allocated buffers. This arrangement affords a number ofadvantages on the design as compared with a single, larger buffer: (1)On general purpose computers, smaller blocks of memory are easier toallocate and move than larger ones; (2) smaller-size numbers (e.g., 16bits for a 64 KB buffer) may be used to keep track of locations in thesmaller buffers; and (3) this arrangement allows a simplified scheme forkeeping track of areas that have been overwritten with fresh data.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows an overall block diagram of the parts of the preferredembodiment and their interrelationships.

FIG. 2 shows the scheme for storing data in the acquisition buffers,using nodes to mark intervals of silence as well as the beginnings andends of buffers. The array of acquisition buffer records is shown alongwith three representative acquisition buffers: one fully recorded, onepartially recorded, and one not yet recorded.

FIG. 3 shows the user interface, comprising the display and usercontrols, which may be embodied in a hand-held device or in a computerprogram.

FIG. 4 is a flowchart of the continuous recording or acquisitionprocess.

FIG. 5 is a flowchart of the periodic process of updating the display asaudio data are acquired.

FIG. 6 is a flowchart of the process of playing back acquired materialin response to a user's request.

FIGS. 7-10 shows four flowcharts, detailing actions taken in response tovarious user controls.

FIG. 11 shows a pocket recording device.

FIG. 12 shows a wrist recording device.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows the major components and their interrelationships: ananalog input transducer such as a microphone feeding a continuous signalto an analog-digital converter, which in turn passes a train of digitalsample values to a set of input buffers. From these the sample train iscontinuously supplied to a circular set of acquisition buffers. Thecontents of the acquisition buffers are shown on a periodically updateddisplay, and parts of the contents of the acquisition buffers may, bymeans of user controls, be played back through an output transducer suchas a speaker or selected for preservation in permanent storage.

Acquisition and Storage of Audio Data

Analog audio data are acquired through a microphone, telephone orsimilar transducer or input, converted into digital data at a certainsampling frequency by an analog-digital converter, and storedsequentially in a circular series of input buffers. (input is by nomeans limited to these sources; it may also include the output ofmusical, medical, seismic or weather instruments. Input may also bedigital in nature, as for example a stream of data broadcast over theInternet or other network, or issuing from any type of digitalinstrument.) In an arrangement typical of audio on personal computers, aperipheral sound card incorporating an analog-digital converter acquiresaudio signals and stores them sequentially via direct memory access(DMA) in a queue of input buffers allocated by an application program.As each input buffer in turn becomes full, an interrupt signal or amessage identifying the particular buffer is generated. When theapplication program receives this signal or message, the sound data aretransferred from that input buffer to acquisition buffers in memory inthe manner described in the following paragraphs. Concurrently with thistransfer of data, audio data continue to be fed by the analog-digitalconverter into another input buffer or buffers, each of which in turngenerates a further interrupt or message when it becomes full. After thecontents of each buffer are transferred to the acquisition buffers, theinput buffer is assigned to the tail end of the queue awaitingrecording. A-minimum of two input buffers, used alternately, isrequired. Particularly in multitasking systems, there may be times whenno input buffers are available for recording because all allocated inputbuffers are still awaiting transcription; when this occurs, new inputbuffers may be allocated. The memory allocated for these buffers may befreed when no longer needed, or when the number of unrecorded butallocated buffers exceeds a certain number.

An array of acquisition buffer records, illustrated in FIG. 2, providesaccess (for recording, playback and transfer to permanent storage) tothe acquisition buffers and to the audio data stored in them. Eachacquisition buffer record contains the following data: tTime Number ofsampling intervals elapsed, at the beginning of this buffer, since thebeginning of the program run. In the record that terminates the array,this value is zero. tEndTime Number of sampling intervals elapsed, atthe end of this buffer, since the beginning of the program run. pWherePointer to the acquisition buffer holding the audio data. In the recordthat terminates the array, this value is null.

The various acquisition buffers each contain, in addition to audio data,at least two nodes, which mark the beginning and the end of each buffer.Additional nodes, interspersed in the audio data, are used to markintervals of silence (as determined in the manner described below)within the buffer. The successive nodes within each acquisition bufferform a two-way linked list. Each node contains the following data:tDuration Duration of silent interval, in sampling intervals. Zero fornodes marking the beginning or the end of a buffer. nNext Number ofsamples of sound data between the end of the current node and thelocation of the next node. nPrev Number of samples of sound data betweenthe beginning of the current node and the end of the previous node. Thisfacilitates reverse playback as well as searching backwards.

FIG. 2 shows three acquisition buffers in various states of beingrecorded: Buffer 10 is completely filled, and has two silence nodesinterspersed in the audio data; these forming a linked list with thebeginning and end nodes of the buffer. Buffer 11 is partially filledwith audio data, and its single silence node is similarly linked withthe beginning and end node. Buffer 10 is empty of data, containing onlythe beginning and end nodes, which also form a linked list.

(In this acquisition buffer scheme, the data are held stationary inmemory and accessed by moving cursors, as described below. Analternative dataflow arrangement is possible, in which data flow througha train of memory elements, all data elements moving to the nextelements as each new sample is read in.)

In the recording process illustrated in FIG. 4, the continuouslyacquired audio data are written successively to acquisition buffers,beginning with the acquisition buffer pointed to by the first record inthe acquisition buffer table. When the recorded data reaches the end ofthat buffer (as indicated by reaching the node there, block 7),recording proceeds by writing to the acquisition buffer pointed to bythe next record in the table. When the recording reaches the end of theacquisition buffer pointed to by the last acquisition buffer record, orthe end of any other acquisition buffer except the last, recordingproceeds with the acquisition buffer pointed to by the first record inthe table, overwriting the existing contents of that acquisition buffer,(blocks 8-9).

On startup, each acquisition buffer is initialized with two nodes, oneat the beginning of the buffer and one at the end. The nNext element ofthe beginning node points to the node at the end, and the nPrev elementof the end node points to the beginning node (block 1; see also FIG. 2):

Any interval of audio data whose absolute amplitude values do not exceeda certain minimal level (the amplitude threshold) over a certain minimalduration (the duration threshold), and whose length exceeds that of anode, is treated as an interval of silence and is represented by a nodein an acquisition buffer.

(This is a simple criterion for detecting an interval of silence; moresophisticated ones are possible, including those involving the detectionand cancellation of ambient noise. Moreover, other signal-enhancementtechniques, such as automatic gain control, may be incorporated in thecontinuous recording process.)

As audio data are transferred from the input buffers to the acquisitionbuffers, the successive amplitude values in the audio stream aremeasured against the amplitude and duration thresholds (FIG. 4, block15), and the stream is accordingly stored in the acquisition buffers assegments of sound data interspersed with silence nodes. Whenever a newsilence node is written in an acquisition buffer, its nPrev member isassigned the number of samples written between this node and theprevious node, and the nNext member of the previous node is assigned thesame value. Similarly, this node's nNext member is assigned the numberof samples that may be written between this node and the node at the endof the acquisition buffer, and the end node's nPrev member is assignedthe same value.

When a sufficient interval of silence has been detected, the recordingcursor goes back to the beginning of the sequence of amplitude values inthe acquisition buffers corresponding to this interval and writes asilence node there (blocks 15-16); this may involve going back into aprevious acquisition buffer. Thus, if the data values at the end of aninput buffer and those at the beginning of the next buffer fall withinthe threshold values for silence, or if the entire contents ofsuccessive buffers fall within the threshold values, a single node isgenerated with the total duration of the consecutive silent intervals.

If the beginning of a “silent” sequence is found to be within less thana node's length before the end node of the relevant acquisition buffer,the remaining spaces are filled in with actual amplitude values and thebeginning node of the next acquisition buffer is used to mark theinterval of “silence”; its duration is set to the remaining duration inthe “silent” sequence.

In the foregoing discussion, references to “next” and “previous”acquisition buffers should be understood in light of the following: Whenthe recording of incoming audio data reaches the end of an acquisitionbuffer, it resumes at the position following the beginning node of theacquisition buffer pointed to by the next acquisition buffer tableelement, except if the present acquisition buffer table element is thelast element in the table. In that case, the “next” acquisition bufferis that pointed to by the element at the beginning of the acquisitionbuffer table. Likewise, the acquisition buffer table element “previous”to that at the beginning of the table is that at the end of the table.In this way, recording of amplitude data proceeds in a circular fashion,overwriting the oldest data with fresh data.

Event-activated recording, as described above, may be effected asfollows: On detection of a suitable triggering event, a predeterminedportion of the current contents of the acquisition buffer (which may bethe buffer's entire contents, or may correspond to a time interval thata user may specify) is transferred to the (permanent) recording mediumnormally used for event-activated recording. This has the effect ofpermanently preserving the events leading up to the triggering event.

Alternatively, the portion of the acquisition buffer contents leading upto the triggering event may be transferred to a second acquisitionbuffer, which will be used in place of the original acquisition bufferfor all purposes described below, including the presentation of data tothe user for selection for recall and/or preservation.

This record of the events leading up to the triggering event isadvantageously placed in the permanent record (or the second acquisitionbuffer) just prior to the triggering event, and advantageously forms acontinuous record with the triggering event and the events following it.It is of further advantage to mark the triggering event in thiscontinuous record. Such a continuous record may be effected by:

(1) Continuing to record in the acquisition buffer the triggering eventand the events following the triggering event,

(2) Concurrently with this continuous recording, copying the appropriateportions of the acquisition buffer contents to the permanent record (orsecond acquisition buffer), preserving the record of events prior to,including, and following the triggering event, and

(3) Recording events directly to the permanent record (or secondacquisition buffer) when the copying process has caught up with thedirect recording process by reaching the present moment.

Permanent preservation of the record of events may alternatively beeffected by:

(1) Continuing to record in the acquisition buffer the triggering eventand the events following the triggering event, and

(2) Reserving the appropriate portions of the acquisition buffercontents (i.e., the record of events prior to, including, and/orfollowing the triggering event) from being overwritten by the continuousrecording process.

Whenever a previously recorded acquisition buffer is about to beoverwritten with fresh data, the block record is checked to see if anyof the buffer's contents have been selected for capture. If so, theselected data are written or appended to a file on the computer's harddisk (block 11). This file may be in a standard sound wave file format,suitable for playing through a standard media player program. Also, atthis time, the origin time (whose value is the earliest time stored inthe acquisition buffers) is updated to equal the start time of the nextacquisition buffer record (block 12). Thus, the contents of anyacquisition buffer in the process of being overwritten are unavailablefor recall or storage.

In general, captured data may be preserved by writing it to anypermanent storage medium, such as a disk, tape or (non-volatile)solid-state memory. Captured data may be preserved at any time betweenthe decision to capture and the time that the data is overwritten;however, deferring storage until the acquisition buffer is about to beoverwritten facilitates giving the user the ability to cancel thedecision to capture a block of data.

When the system is to be shut down for any reason (as, for example, acritically low battery level), all captured data in the acquisitionbuffers are preserved prior to shutdown.

The capture interval, i.e., the time interval that may be capturedbefore the user's decision to record, is a function of the quantity ofrecording medium and the recording density (i.e., length of recordingtime stored in a given quantity of recording medium). If captured data,rather than being transferred out of the acquisition buffer, are storedin a newly reserved area of the acquisition buffer, then the captureinterval will diminish as this area becomes filled with captured data.Compression algorithms may be employed concurrently with recording thesound (as with the silence-node scheme described above), and/or oncaptured data in permanent storage, in order to maximize both thecapture interval and the total recording time capacity of the device. Inaddition to data compression, captured data may be further processed forquality or compactness, for example via phonetic transcription or anysignal-enhancing postprocessing method. (Within practical limits, thesetechniques may be used on the continuously recorded data as well.)

User Interface

The user is provided with a display representing audio data currently inthe acquisition buffers, as well as controls to allow the user to recall(play back) data and to mark data for capture or permanent storage. Allof these are shown in FIG. 3.

The display consists of three parts:

(1) A horizontal time scale marked in minutes and/or seconds,representing the positions in time relative to the current time of datain the acquisition buffers. (Alternatively, this scale may be markedwith absolute time-of-day markings.)

(2) Parallel to the time scale, the amplitude graph, representing aseries of numeric values, each of which is the maximum (or the averageor some other relevant) amplitude value over its corresponding timeinterval. The amplitude graph may be a bar graph, a series ofone-pixel-wide vertical lines whose heights correspond to the successivevalues in the amplitude record; or it may be a density graph, a seriesof vertical lines each drawn where the sum of successive amplituderecord values reaches a fixed value.

(3) Parallel to the time scale and the amplitude graph, the block graph,representing the division of the contents of the acquisition buffersinto blocks for selection and capture by the user. The displayed blockgraph represents the contents of the acquisition buffers divided intoblocks separated by intervals of silence that exceed a minimum duration(the block separation threshold). The minimum duration of a block isidentical to the duration threshold discussed under “Acquisition andStorage of Audio Data.” Uncaptured blocks are shown as hollow rectangleson the block graph; captured blocks are shown as solid rectangles; andintervals of silence are shown as blank spaces between the displayedblocks.

A speaker, earphone or similar transducer or line output is provided forthe purpose of playing back data in the acquisition buffers.

A cursor is provided, in the form of a vertical line intersecting orabutting the time scale, the block graph and the amplitude graph, toindicate (while in playback mode) the position on the displaycorresponding to the data being (or about to be) played back.

A function (the display update function) is called from time to time inthe course of continuous recording, to update the display and itsunderlying data; this function is flowcharted in FIG. 5 and described indetail below.

The three parallel parts of the display represent, at any given time,the entire contents of the acquisition buffers (less any acquisitionbuffer currently being overwritten). As the physical space given to thedisplay may be insufficient to represent these contents, the display maybe provided with a means of scrolling its view to show the data for alltimes between the earliest time stored (the origin time) and the currenttime. Additionally, intervals of silence that exceed a certain length(the minimum ellipsis length) may be represented in all three displayparts as the minimum ellipsis length, with a mark to show a “break” inthe display.

Controls and Operation

Controls (buttons) are provided for manipulating the cursor, playingback portions of recorded blocks, and capturing blocks: Move Back Ifheld and released quickly: Moves the (FIG. 7) cursor to the beginning ofthe current block. If at the beginning of a block, moves to thebeginning of the previous block. If not in default mode, the cursorappears at the beginning of the most recent block. If held down: Movesthe cursor continuously backward within the current block and previousblocks, as long as it is held down. (Optionally, plays back in reversethrough this.) Normal playback continues when the button is released. Ifnot in playback mode, either action initiates playback mode as soon asthe button is released. Move If held and released quickly: Moves theForward cursor to the beginning of the next block to (FIG. 8) the right.If the cursor is on the rightmost block, this has the same effect aspressing Stop. If held down: Plays fast-forward through the currentblock and succeeding blocks as long as the button is held down. Playbackat normal speed continues when the button is released. Stop Stopsplayback and resumes continuous recording (i.e., default mode). Thecursor disappears. Capture Toggles the capture state of the currentblock (FIG. 9) (as indicated by the visible cursor). May be pressed atany time to capture the current block. A second press releases thecurrent block from capture. In default mode, captures the most recentblock(s). (See below for details of default mode behavior.) Capture AllWhen pressed, all blocks acquired thereafter are marked for capture;when pressed a second time, restores normal mode in which blocksacquired thereafter are not marked for capture. In either mode, theCapture control may be used to toggle the capture state of any.particular block, including the one currently being acquired. SplitDivides the current block into two blocks. The (FIG. 10) division ismade at the last silence before the current cursor position that exceedsa certain minimum duration (the split duration threshold) before thebutton is pressed. A second press undoes the division, if no otherbutton has been pressed after the Split button (or if a sufficient timehas elapsed from pressing the Split button).

The behaviors described above apply to playback mode, which is enteredby pressing or holding down the Move Back button, then releasing it.While in playback mode, the system continuously plays back material inthe acquisition buffers, in a forward direction at the same speed asrecorded, as long as no keys are being pressed or held down. Theplayback process is illustrated in FIG. 6.

The default mode of the device goes into effect on startup, and isreverted to from playback mode either on pressing the Stop button orwhen playback reaches the current time (or, if the implementation iscapable of simultaneous recording and playback, after the controls havenot been manipulated for a certain interval of time). In default mode,no cursor is shown on the display; pressing the capture button capturesthe most recent block. (Alternatively, pressing Capture once may capturethe most recent minute, pressing it twice in rapid succession maycapture the most recent two minutes, and so on. These intervals may beadjusted to the nearest block boundary.) If this most recent blockcontinues through the time the capture button is pressed—i.e., if lesstime than the block separation threshold of detected silence had elapsedbefore the capture button was pressed—the device continues to capturedata until either the capture button is pressed again or a sufficientpause has been detected. (The device may sound an audible warning atthat point, giving the user a choice whether to continue capturing.)

In some implementations (e.g., on personal computer systems employingtwo sound cards), the acquisition of current audio data through theinput buffers into the acquisition buffers continues through the user'smanipulation of the controls and during playback. In otherimplementations, the continuous recording is suspended while in playbackmode. Specifically, a typical single sound card on a personal computercannot record and play back simultaneously.

Data Elements

The data representing intervals of sound and silence are identified bythe times of their occurrence and the Places where they are stored inthe acquisition buffers. Times are identified by unsigned long integersrepresenting the number of sample intervals (or “ticks”) that haveelapsed in the course of the continuous recording process. In workingwith these quantities, consideration must be made for the possibility ofthe maximum possible long integer value being reached and “wrappingaround” to zero; this may be handled by an appropriate overloading ofoperators and functions, taking the current value of origin time intoaccount.

Places in the acquisition buffers are represented by structurescontaining the following data: tTime Number of sample intervals elapsedup to this point in the continuous recording process. pAcqbuff Pointerto the acquisition buffer record for the acquisition buffer containingthis Place. pWhere Pointer to the current place in the acquisitionbuffer. If in a silence node, this points to the beginning of the node.tTixTillNode Number of samples (going forward) before the next node; 0if in a node. tNodeTix Number of sample intervals elapsed within thecurrent silence node; 0 if not in a node, so count begins with 1 andends with the node's tDuration.

Places constitute the base class for a number of logical cursors, forthe various functions of recording, playing back and storing sampledaudio data, as they allow random access to any point in the acquisitionbuffers. For the same reason, places are also incorporated in the dataunderlying the user interface, where they allow for easy translationbetween the display elements and the corresponding places in the userinterface graphs.

Implementation

The data underlying the user interface are implemented in the form of acircular, expandable array of pixel records each containing sufficientdata to support one linear element of the time scale, the amplitudegraph, and the block graph. Each pixel record contains the followingdata: nBlocking An enumerator indicating the status of this element inthe block graph. Possible values are: SPACE (indicating an interval, orpart of an interval, of silence), BLOCK (indicating an interval, or partof an interval, of sound), and CAPTURED_BLOCK (indicating an interval,or part of an interval, of sound that has been marked by the user forpermanent storage). sAmplitude A sample value representing the amplitudeto be shown for this element in the amplitude graph. plWhere A Placerecord corresponding to the beginning of the corresponding interval inthe acquisition buffers. At the beginning of a contiguous sequence ofBLOCK, SPACE, or CAPTURED_BLOCK pixels, this is the location at whichthe actual interval of silence or sound begins in the acquisitionbuffers: it points to a node (for SPACE) or points to data immediatelyfollowing a node (for BLOCK or CAPTURED_BLOCK).

FIG. 5 illustrates the display update function, which writes new pixelrecords and is called at regular intervals in the continuous recordingprocess, corresponding to the duration of one pixel record. Itsarguments include the data for the pixel record it writes: the currenttime and Place in the acquisition buffer and the relevant (maximum oraverage) amplitude value; this value is reset in preparation for thenext pixel interval. The display is updated to show the new pixel, inall three graphs, at the right edge of the display (representing thepresent time), shifting all previous pixels to the left (blocks 26-27).To facilitate this shifting in the display, a pointer to the currentpixel (representing the current time) is incremented with each new pixelwritten (block 21).

Each new pixel record in turn is written at the location in the pixelarray immediately following the last written pixel element. When the endof the pixel element array has been reached, the next pixel element iswritten at the beginning of the pixel array, provided that the existingpixel element at that location does not represent data remaining in theacquisition buffers. If the existing pixel record does representstill-valid data (block 22), then the pixel array is expanded (as byreallocating memory at a different location).

The time scale is constructed from time values contained in the plWheremembers of each pixel record, which will not necessarily form a linearprogression of times: Not only are the Places contained in pixel recordsadjusted on block boundaries, but there may also be breaks representinglong intervals of silence, or places where an extra pixel was added toproduce a visible blank space in the block graph for an otherwiseinsufficiently long interval of silence. The time scale is appropriatelymarked to reflect these possible irregularities.

Any interval of silence exceeding a predetermined length (the minimumellipsis length) is represented by an “ellipsis” or break in thedisplay, shorter than the corresponding actual number of pixels. Whensuch an interval is encountered in playback, a short “ellipsis” waveformrepresenting the break is played instead of playing through the entireduration of the silence.

Playback is achieved by means of a scheme analogous (in reverse) to therecording scheme: The Place represented by the current pixel record (asindicated by the current cursor position) is found in the acquisitionbuffer, and the audio data following it is transcribed to a queue ofoutput buffers allocated by the application program. In thetranscription process, audio data are transcribed identically andsilence nodes are expanded to their full durations; the intervals storedas silence nodes may be filled with zero-amplitude values or with adithering waveform containing values differing from zero by no more thanthe amplitude threshold (FIG. 6, block 37). Output buffers are sent, inthe order transcribed, to a waveform output device, which in turn routestheir contents (via direct memory access (DMA)) through a digital-analogconverter to the audio output. Concurrently with this playback process,audio data continue to be transcribed from the acquisition buffers intoanother output buffer or buffers, each of which in turn generates afurther interrupt or message when it has finished playing back. Whenthis occurs, the output buffer is assigned to the tail end of the queueawaiting transcription from the acquisition buffers. (By keeping trackof the corresponding Places in the acquisition buffers and the displaydata, the cursor may be appropriately advanced on receiving this messageor signal.)

A minimum of two output buffers, used alternately, is required.Particularly in multitasking systems, there may be times when no outputbuffers are available for transcription because all allocated outputbuffers are still awaiting playback; when this occurs, new outputbuffers may be allocated. The memory allocated for these buffers may befreed when no longer needed, or when the number of allocated butuntranscribed buffers exceeds a certain number.

When any key that moves or activates the cursor is pressed, playbackmust be interrupted. This involves forcing the return of any partiallyplayed output buffer and removing any remaining output buffers from boththe queue awaiting playback and the queue awaiting transcription fromthe acquisition buffers (FIG. 6, block 46). Responses to various usercontrol actions are flowcharted in FIGS. 7-10.

In implementations that are not capable of concurrent recording andplayback, recording must be terminated in the same instantaneous mannerwhen any key that moves or activates the cursor is pressed. Thisinvolves forcing the return of any partially recorded input buffer(noting the actual quantity of recorded data therein) and removing anyremaining input buffers from both the queue awaiting recording and thequeue awaiting transcription into the acquisition buffers.

In implementations capable of concurrent recording and playback,processing time must be appropriately apportioned (by means of programthreads or otherwise) to the queue of input buffers awaitingtranscription to the acquisition buffers and the queue of output buffersawaiting transcription from the acquisition buffers, when both processesare actually occurring at the same time. That is, processing time shouldbe apportioned to minimize the number of buffers in the input queuewhile ensuring that there will be a certain minimum number of buffers inthe output queue.

The present invention may be used to record telephone conversations andmake portions of those conversations available for recall orpreservation to parties to the conversation upon the consent of any orall of the other parties. In this context, it is advantageous:

(1) to generate an indication audible to all (or selected) parties whenany party moves to recall or preserve a portion of the conversation,

(2) to replay audibly for all parties (or for a selected party orparties) any portion of the conversation that any party has selected forrecall or preservation,

(3) to require the consent or all parties (or of a selected party orparties) to the recall or preservation of any portion of theconversation before effecting that recall or preservation,

(4) to authenticate the identity of parties to the conversation (for thepurpose of consent, or any other purpose) using “digital signatures” orother cryptographic means, or biometric means such as “voiceprint,” and

(5) to clear the contents of the acquisition buffer(s) when any party(or any selected party) has left the conversation, or on the request ofany party, thereby ensuring the security of this consent arrangement.

The operations of initiating recall, controlling playback and effectingpreservation of portions of a telephone conversation may be implementedusing a display and controls similar to those described above, or moresimply using well-known voice response methods and telephone keypadbuttons.

Effective after-the-fact recording of a telephone conversation, evenwith this type of security arrangement, need not be limited to theduration of a call; any party may initiate a call to a recording servicewhich will in turn contact the other required parties and secure theirconsent. The acquisition buffers may be located either centrally or inthe local telephone equipment of any of the parties; for added security,their contents may advantageously be encrypted. An encrypted portion ofthe acquisition buffer's contents may advantageously be offered to anyparty to a conversation for review and/or file download to a computer orappliance on the Internet.

In the following discussion:

(1) The phrase “interim record” or “interim recording” refers to thecontinuous recording taken directly from a conversation, or (asappropriate) the file, medium or media containing the same, or portionsof the same.

(2) The verb “to conceal” (or the noun “concealment”) refers to any andall such methods of making the interim record inaccessible to users,including encryption.

(3) The verb “to disclose” (or the noun “disclosure”) refers to any andall methods of making the interim record accessible to users, includingdecryption.

(4) The term “disclosed record” refers to any record of all or part ofthe conversation that has been disclosed to a user or party.

(5) The phrase “ordering party” refers to the party first requestingdisclosure of the interim record, and “counter-party” refers to anyother party to the conversation. (The ordering party may or may not bethe same party subscribing to a service that includes the presentinvention.) The more general term “user” refers to parties,advantageously authorized by a party to the conversation, to whom theall or part of a conversation may be made accessible.

The present invention admits of a considerable number of implementationmethods with regard to (a) recording or compiling the interim record,(b) concealing the interim record, (c) informing parties of therecording and securing their consent, (d) applying such consent toportions of the conversation or of the interim recording for the purposeof disclosing the same, (d) making accessible, transferring or otherwisedisclosing the interim record, (e) enabling users and parties to reviewdisclosed records, and (f) charging users and parties for the use of thepresent invention. Various implementation elements with regard to theseconsiderations are enumerated in the following paragraphs; combinationsand permutations of these elements, each with particular advantages, aswell as specific methods of implementing the same using well-knownequipment, media, algorithms, strategies and techniques, will occur tothose skilled in the art.

The interim recording of a conversation may be compiled and stored in anumber of different places:

(1) On or in an appliance on a user's premises, vehicle or person,connected or advantageously integrated with the telephone, or in thetelephone itself.

(2) On or in equipment under the control of the telephone company orother entity providing the service.

(3) On a remote file server, computer, or similar device, or anywhere onthe Internet or a similar network of interconnected computers and/orappliances. Such a device may be located anywhere on earth or on asatellite.

It is advantageous to have the interim recording take the form of acircular acquisition buffer as described above. Not only does thisconserve storage space (of advantage where storage capacity is limited,as in a handheld device or where a large number of customers must beserved); it also assures that (concealed) records of conversations thathave not been disclosed are eventually destroyed. However, the interimrecord need not take the form of a circular buffer. Instead, it may takethe form of a file or a series of files, with such files advantageouslybeing preserved or destroyed as desired by appropriate users or by theparty providing the service.

It is important to make the interim recording inaccessible to all usersand conversation parties unless and until appropriate consent has beensecured. Encryption is advantageous where the recording is accessible tosuch users or parties—for example, in the form of a file on a computer,in removable memory or through a data transfer port. However, encryptionmay not be necessary; it may be suffice simply to make the interimrecording inaccessible—for example, by locating it in inaccessible,non-removable memory in a telephone or appliance, or on a remotecomputer or device inaccessible to users and conversation parties.

Parties to the conversation may advantageously be identified (or atleast distinguished from one another and matched to their portions of aconversation, responses to prompts, and the like) using well-knownalgorithmic methods (e.g., “voiceprint”). Such identification may beperformed continuously as the interim record is compiled, or it may beperformed as required at any time prior to or at disclosure of therecord to a particular party or parties. Identities of partiesidentified by such means, and their relationships to parts of theconversation, are advantageously preserved in the interim and/ordisclosed records, as appropriate, to avoid having to apply an algorithmmultiple times to the same material.

Consent may be applied in a number of ways:

(1) Each party may be automatically granted the right to disclose thatparty's part of the conversation to him- or herself. (For security, suchparty may advantageously be required to authenticate his or her identityby means of a password or cryptographic key, or by algorithmicallymatching that party's voice to parts of the conversation.)

(2) The consent of each party may be required for any other party todecrypt that specific party's portion of the conversation(advantageously matched by algorithm to that party's voice).

(3) Alternatively, the consent of all parties may be required to decryptthe whole conversation or any portion of it.

A number of different methods may be used to enable the disclosure,review and preservation of the (disclosed) conversation:

(1) Consent may be secured in the course of the conversation, coveringeither the portion of the conversation preceding the point at whichconsent is given or the remainder of the conversation, or both. Suchconsent may be secured, at the beginning or at any point in theconversation, by the ordering party speaking a predetermined word orphrase, or pressing a predetermined key or key sequence. This willinitiate a mechanized voice prompt, asking for the consent (in turn) ofany party to the conversation that cares to give thesame—advantageously, once for consent to the past portion of theconversation and once for the future part. The voices responding tothese prompts may be advantageously matched to other portions of theconversation.

(2) Alternatively, a party to the conversation may initiate (by similarmeans) a mechanized voice prompt that warns the parties that theconversation may be recorded. That party (and possibly other parties)would thereby be enabled to decrypt, review and/or preserve a record ofthe conversation (as least the portion following the warning) withoutrequiring the consent of the other parties.

(3) Consent may be secured after the end of the conversation byrequiring the user to contact by telephone (or generating an automatedtelephone call to) each appropriate counter-party (whose telephonenumbers have been recorded together with the conversation) in which arecorded or mechanized voice identifies the conversation by somecombination of date, time and telephone numbers (and, if appropriate,names) of parties and requests the counter-party's consent to making theconversation.

(4) Consent may be secured by telephone by requesting the appropriateuser to press telephone keys or verbally responding (“yes” or “no,” orotherwise as appropriate) to an appropriate voice prompt. The voicecharacteristics of the speaker's responses to automated questions may beeffectively matched by algorithm to portions of the recorded telephoneconversation. Passwords (sometimes known as PINs, or personalidentification numbers) may also be required and used (advantageouslyemploying well-known methods) to authenticate the identity of parties.

(5) Consent may also be secured on the Internet. This may involve any ofseveral well-known methods of securing and authenticating the identityof parties, involving passwords, certificates, secure socketconnections, keys, private-public key exchanges, or the like.

(6) The consent of the parties needed to review or play the conversationmay be secured, and the interim record appropriately disclosed, eitherbefore or after a file containing the interim record (on a remotecomputer, device or network) is downloaded by a user. Disclosure of theinterim record following download may be effected by supplying the userwith a cryptographic key to disclose the record. In the case ofmulti-party conversations, a cryptographic key for each party (providedupon that party's consent) will advantageously be required for otherparties decrypt the conversation.

(7) Where parties are required to exchange cryptographic keys,well-known methods (such as the use of public-private key pairs, inpossible combination with concealed symmetric keys) may beadvantageously employed to conceal keys from and/or restrict their useby users.

Disclosure may be effected in any of the following ways:

(1) Where the interim record resides in inaccessible memory (as on acellular telephone, for example), all or part of the interim record maybe transferred to removable memory or (through a data port) to anotherdevice, advantageously in the form of a file. Such transfer may beenabled prior to securing consent, by transferring an encrypted record,transferring the record to inaccessible memory or otherwise keeping therecord concealed. On securing consent, disclosure may be effected bysupplying the user with a cryptographic key to disclose the record. Inthe case of multi-party conversations, a cryptographic key for eachparty (provided upon that party's consent) will advantageously berequired for other parties decrypt the conversation.

(2) An encrypted portion, or the whole, of the acquisition buffer'scontents may advantageously offered to any party to a conversation forreview and/or download (say, as a file or files) to a computer, storagedevice or similar appliance from a network such as the Internet. Oncedownloaded, the file may be decrypted on securing the appropriateparties' consent.

Users may be enabled to review disclosed records by any of the followingmethods:

(1) Listing all conversations in the interim record, advantageously bydate, time and parties, and enabling a user to select a particularconversation for review, obtaining other parties, consent, disclosure,preservation and the like. Such listing may be implemented using voiceprompts, or advantageously using a display on a telephone or appliance,or in a computer program or browser application.

(2) Using well-known controls (including, where appropriate, telephonekeys) to play, pause, and/or skip forward and/or back through adisclosed record, and (advantageously) to select portions of thedisclosed record for preservation, advantageously in conjunction withvoice and/or text prompts, advantageously according to well-knownmethods.

(3) Providing a user interface comprising a display and controls similarto that disclosed above and in FIG. 3, advantageously identifyingdistinct parties to the conversation by means of distinct colors,hatchings or the like.

(4) Using an appropriate computer program or browser script, as on theInternet.

An appropriate user (e.g., the owner of an appliance containing aninterim record) may be advantageously enabled to clear or deleteconversations or portions thereof from the interim record,advantageously with confirmation of any attempt to delete them.

Users of the present invention may be charged for its use in any thefollowing ways:

(1) A telephone and/or appliance (or an upgrade or component of thesame) may be sold to a user for charges including a one-time usage orlicense fee.

(2) A user may be charged a periodic subscription fee for the service.Such subscription fees may advantageously vary with the quantity ofstorage used. Alternatively, the service may be offered with someminimal storage configuration for free, with fees charged fordisclosing, downloading or the like.

(3) A user may be charged a fee for each instance disclosing an interimrecord to him- or herself or to any other party, or downloading a file,or the like. Alternatively, a user may be enabled to purchase a“packages” allowing a specific number of disclosures, downloads and/orsimilar actions.

(4) Parties to a conversation may possibly be enabled to charge fees toother parties for their consent.

(5) Payment of any fees may be effected by well-known methods, includingcredit card and online methods (e.g., PayPal).

In applications of the present invention, a non-volatile,interchangeable memory module (as of the type commonly known as “flashmemory”) may be used as a recording medium, for the acquisitionbuffer(s) and/or as a permanent storage medium. Where suchinterchangeable memory is used, it is advantageous to use such memoryinterchangeably in a variety of devices and/or to record a variety ofsources of continuously recordable events (e.g., radio, television, livemicrophone, weather instruments, medical instruments, musicalinstruments). In this way a single flash memory may be used to acquire,recall and/or preserve data from various sources.

This is effected by recording the identity of the various sources of thedata in said flash memory, so that the data may be appropriatelyreproduced. The present invention may be advantageously embodied in aportable device that clips to a pocket or other article of clothing, asshown in FIG. 11. A microphone, camera, camera lens, infrared orradio-frequency receiver, or similar device is attached to the clip,exposed to the outside environment. The clip may also carry aphotoelectric cell or other ambient power source exposed to theenvironment, drawing power from ambient light (or other ambient power),advantageously charging a storage battery so as to provide continuouspower to the apparatus. The recording device advantageously records todigital memory or a similar medium with modest power requirements.

In this arrangement, the clip physically stabilizes the receivingdevice, placing it in an advantageous position to record ambient eventssurrounding a person while avoiding extraneous noises such as mightresult from handling the microphone or allowing it to “flop around”freely. A display and controls for effecting recall and/or preservationof recorded data, similar to those described above, may beadvantageously provided on the portion of the device carried in thepocket.

Here it is of further advantage to preserve material by reservingportions of the buffer from being overwritten, in order to use theentire memory store as needed. To effect transfer of material selectedfor permanent preservation out of the device, a data transfer port(infrared, Toslink or the like) should be provided. Alternatively, flashmemory (as described above) may be used for this purpose.

An alternative embodiment of the present invention is a small devicemounted on a wrist band or bracelet to secure the recording device to ahuman wrist, in the manner of a wristwatch. This device advantageouslyincorporates an attached microphone or microphones, exposed to theoutside environment so as to pick up ambient sounds. This microphone ormicrophones may be recessed below a projecting lip on the upper surfaceof the recording device, so as to pick up ambient sounds while avoidingnoise from clothing objects that may come into contact with the surfaceof the recording device. The top surface of the device provides adisplay, similar to that described above, representing events recordedand/or to be recorded, as well as controls for effecting recall andpreservation of events. Display means may be implemented as an LED orLCD display, and controls may be implemented as discrete buttons and/oras touch-sensitive regions of said display, advantageously manipulatedusing a stylus.

The wrist band physically stabilizes the microphone, placing it in anadvantageous position to record ambient events surrounding a personwhile avoiding extraneous noises such as might result from handling themicrophone or allowing it to “flop around” freely.

The device may also carry a photoelectric cell or other ambient powersource exposed to the environment, drawing power from ambient light (orother ambient power), advantageously charging a storage battery so as toprovide continuous power to the apparatus. The recording deviceadvantageously records to digital memory or a similar medium with modestpower requirements.

Here again it is of further advantage to preserve material by reservingportions of the buffer from being overwritten, in order to use theentire memory store as needed. To effect transfer of material selectedfor permanent preservation out of the device, a data transfer port(infrared, Toslink or the like) should be provided. Alternatively, flashmemory (as described above) may be used for this purpose.

This apparatus may also be advantageously combined with or incorporatedin a wristwatch and/or any of the various devices or functions currentlyor hereafter combined with wristwatches: In this case, the recordingdisplay is advantageously implemented as one of a plurality of availabledisplay modes of the device.

The recording device advantageously records to digital memory or asimilar medium with modest power requirements.

Although the present invention has been described in connection withparticular applications thereof, it is to be understood that additionalembodiments, applications, and modifications, which will be obvious tothose skilled in the art, included in the spirit and scope of theinvention.

In the following claims:

The term “events” includes, without limitation, any auditory or visualevents, data or signals, still or moving pictures, machine events,states or signals, and the states, signals or sounds of any musicalinstrument, any of the foregoing being of a discrete or a continuousnature.

The term “recording medium” or “medium” refers to any material orapparatus, whether now existing or hereafter invented, that is capable(whether alone or in combination with other components) of receivingimpressions of events external to said material or apparatus, and ofreproducing said impressions in combination with appropriate means. Thisterm includes, but is not limited to, magnetic and optical media (tapes,disks, etc.) as well as all varieties of electronic memory, bothvolatile and non-volatile.

The term “acquired data” or “recorded data” or “data” refers to saidimpressions of external events on a recording medium.

The term “interval of silence” refers to any time interval ofsufficiently low content or change (as determined by an apparatus oralgorithm) as to be considered empty of any material worthy ofrecording.

The phrase “interim record” or “interim recording” refers to thecontinuous recording taken directly from a conversation, or (asappropriate) the file, medium or media containing the same, or portionsof the same.

The verb “to conceal” (or the noun “concealment”) refers to any and allsuch methods of making the interim record inaccessible to users,including encryption.

The verb “to disclose” (or the noun “disclosure”) refers to any and allmethods of making the interim record accessible to users, includingdecryption.

The term “disclosed record” refers to any record of all or part of theconversation that has been disclosed to a user or party.

The phrase “ordering party” refers to the party first requestingdisclosure of the interim record, and “counter-party” refers to anyother party to the conversation. (The ordering party may or may not bethe same party subscribing to a service that includes the presentinvention.) The more general term “user” refers to parties,advantageously authorized by a party to the conversation, to whom theall or part of a conversation may be made accessible.

While the preferred embodiment of the invention has been described indetail, modifications and adaptations may be made thereto, withoutdeparting from the spirit and scope of the invention, as delineated inthe following claims:

1. A process for recording telephone conversation, said processcomprising the steps of: compiling a concealed interim record of saidtelephone conversation on a buffer comprising at least one finite extentof a recording medium; providing a continuous acquisition means forrecording telephone conversation events in said acquisition buffer;requesting and securing the consent of at least one party to theconversation; and disclosing those parts of said concealed interimrecord according to said secured consent.